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SUMMARY 


Principles for the design of dynamic spatial instruments for communicating quantitative infor- 
mation to viewers are considered through a brief review of the history of pictorial communication. 
Pictorial communication is seen to have two directions: 1) from the picture to the viewer and 
2) from the viewer to the picture. Optimization of the design of interactive instruments using pic- 
torial formats requires an understanding of the manipulative, perceptual, and cognitive limitations 
of human viewers. 


PICTURES 


People have been interested in pictures for a long time (fig. 1). This interest has two related 
aspects. On one hand we have an interest in the picture of reality provided to us in bits and pieces 
by our visual and gross body orienting systems-and their technological enhancements. Indeed, 
Western science has provided us with ever clearer pictures of reality through the extension of our 
senses by specialized instruments. 

On the other hand, we also have an interest in pictures for communication, pictures to transmit 
information among ourselves as well as between us and our increasingly sophisticated information- 
processing machines. This second aspect will be our prime focus, but some discussion of the first 
is unavoidable. 

It is useful to have a working definition of what a picture is and I will propose the following: 

A picture is produced through establishment of a relation between one space and another so that 
some spatial properties of the first are preserved in the second, which is its image. A perspective 
projection is one of many ways this definition may be satisfied (fig. 2). 

The definition may be fleshed out, as cartographers do, by exactly stating what properties are 
preserved, but the basic idea is that, though the defining relation of the layout of the picture may 
discard some of the original information, this relation is not arbitrary. The challenge in the design 
of a picture is the decision what to preserve and what to discard. 

Artists, of course, have been making these decisions for thousands of years, and we can learn 
much from this history. One curious aspect of it, one that I certainly found strange when I learned 
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of it, is that early art was not focused on the preservation of spatial properties that I have asserted 
to be the essence of a picture. 

As art historians have pointed out, early art was often iconographic, depicting symbols, as 
these Egyptian symbols for fractions illustrate, rather than aspiring to three-dimensional realism 
(fig- 3) (Gombrich, 1969). This early history underscores a second aspect of pictures which we 
must consider: their symbolic content. Because of the potentially arbitrary relation between a 
symbol and what it denotes, a symbol itself is not a picture. Symbols, nevertheless, have from the 
very beginning wormed their way into many pictures, and we now must live with both the sym- 
bolic and geometric aspects of pictorial communication. Furthermore, the existence of the sym- 
bolic content of the picture has the useful role of reminding the viewer of the essentially duplicitous 
nature of a picture since, though it inherently represents an alternative space, it itself is an object 
with a flat surface and fixed distance from the viewer. 

The third basic element of pictorial communication is computational. The picture must be cre- 
ated. In the past the computation of a picture has primarily been a manual activity limited by the 
artist's manual dexterity, observational acumen, and pictorial imagination. The computation has 
two separable parts: 1 ) the shaping and placement of the components of the image, and 2) the ren- 
dering, that is, the coloring and shading of the parts (fig. 4). 


While this second part is clearly important and can contribute in a major way to the success of a 
picture, it is not central to the discussion I wish to develop. Though the rendering of the image can 
help establish the virtual or illusory space that the picture depicts and can literally make the subject 
matter reach out of the picture plane, it is not the primary influence on the definition of this virtual 
space. Shaping and placement are. These elements reflect the underlying geometry used to create 
the image and determine how the image is to be rendered. By their manipulation artists can 
define — or confuse — the virtual space conveyed by their pictures. 

While the original problems of shaping, positioning, and rendering still remain (figs. 5 
and 6), the computation of contemporary pictures is no longer restricted to manual techniques. 

The introduction of computer technology has enormously expanded the artist's palette, and pro- 
vided a new 3D canvas on which to create dynamic synthetic universes; yet the perceptual and cog- 
nitive limits of the viewers have remained much the same. Thus, there is now a special need for 
artists, graphic designers, and other creators of pictures for communication to understand these 
limitations of their viewers. Here is where the scientific interest in the picture of reality and the 
engineering interest in the picture for communication converge. 


SPATIAL INSTRUMENTS 


In order to understand how the spatial information presented in pictures may be communicated, 
it is helpful to distinguish between images which may be described as spatial displays and those 
that were designed to be spatial instruments. One may think of a spatial display as any dynamic, 
synthetic, systematic mapping of one space onto another. A picture or a photograph is a spatial 
display of an instant of time (fig. 7). A silhouette cast by the sun is not, because it is a natural phe- 
nomenon not synthesized by humans. 
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A spatial instrument, in contrast, is a spatial display that has been enhanced either by geomet- 
ric symbolic, or computational techniques to ensure that the communicative intent of instrument is 
realized A simple example of a spatial instrument is an analog clock (fig. 8). In a clock the angu- 
lar positions of the arms are made proportional to time, and the viewer’s angle-estimation task is 
assisted by radial tic marks designating the hours and minutes. 

A second aspect of the definition of a spatial instrument, which the clock example also illus- 
trates, is that the communicated variable — time— is made proportional to a spatial property ot the 
display, such as an angle, areas, or length and is not simply encoded as a character string. 

The spatial instruments on which we wish to focus attention are generally interactive. That is 
to say, the communicated information flows both to and fro between the viewer and the instru- 
ment. ’ Some of this bidirectional flow exists for practically all spatial instruments, since movement 
of the viewer can have a major impact on the appearance of the display. However, the displays 1 
wish to consider are those incorporating at least one controlled element, such as a cursor, which is 
used to extract information from and input information to the instrument. 

Spatial instruments have a long history. One of the first ever made, dating from 60-80 BC, 
was an astrolabe-like device uncovered in 1901 near Antikythera, Greece. However, it was not 
fully described until the late '50’s by De Solla Price (1959), who was able to deduce much of its 
principles of operation by x-raying the highly corroded remains (fig. 9). Here the communicated 
variables were the positions of heavenly bodies. Nothing approaching the complexity ot this 
device is known until the 16th Century. It represents a highly sophisticated technology otherwise 
unknown in the historical record. 

Though many subsequent spatial instruments have been mechanical and, like the Prague town 
hall clock (fig. 8), have similarly been associated with astronomical calculations (King, 1978), this 
association is not universal. Maps, when combined with mechanical aids for their use, certainly 
meet the definition of a spatial instrument (fig. 10). The map projection may be chosen depending 
upon the spatial property of importance. For example, straight-line mapping of compass courses 
(rhumb lines), which are curved on many maps, can be preserved in Mercator projections 
(Dickinson, 1979; Bunge, 1965). Choice of these projections illustrates a geometric enhancement 
of the map.' The overlaying of latitude and longitude lines illustrates a symbolic enhancement 
(figs. 11-13). But more modem media may also be adapted to enhance the spatial information t lat 
they portray, as illustrated by the reference grid used by Muybridge in his photographs 
(Muybridge, 1975) (fig. 14). 

Contemporary spatial instruments are found throughout the modem aircraft cockpit (fig. 15), 
the most notable probably being the attitude direction indicator which displays a variety of signals 
related to the aircraft's attitude and orientation. More recent versions of these standard cockpit 
instruments have been realized with CRT displays, which have generally been modeled after their 
electromechanical predecessors (Boeing, 1983). But future cockpits promise to look more like 
offices than anything else (fig. 16). In these offices the computer graphics and CRT display 
media, however, allow the conception of totally novel display formats for totally new, demanding 
aerospace applications. 

For instance, a pictorial spatial instrument to assist informal, complex, orbital navigation in the 
vicinity of an orbiiing spacecraft has been described (ftg. 17) (see also Paper 37, Grunwald and 
Ellis, 1988). Other graphical visualization aids for docking and orbital maneuvering, as well as 
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other applications, have been demonstrated by Eyles (1986) (see also Paper 36). These new 
instruments can be enhanced in three different ways: geometric, symbolic, or computational. 


GEOMETRIC ENHANCEMENT 


In general, there are various kinds of geometric enhancements that may be introduced into spa- 
tial displays, but their common feature is a transformation of the metrics of either the displayed 
space or of the objects it contains. A familiar example is found in relief topographic maps for 
which it is useful to exaggerate the vertical scale. This technique has also been used for experi- 
mental traffic displays for commercial aircraft (fig. 18) (Ellis, McGreevy, and Hitchcock, 1987). 

Another type of geometric enhancement important for displays of objects in 3D space involves 
the choice of the position and orientation of the eye coordinate system used to calculate the projec- 
tion (fig. 19). Azimuth, elevation, and roll of the system may be selected to project objects of 
interest with a useful aspect. This selection is particularly important for displays without stereo- 
scopic cues, but all types of displays can benefit from an appropriate selection of these parameters 
(Ellis et al., 1985; see also Paper 30, Kim et al., 1987). 

The introduction of deliberate spatial distortion into a spatial instrument can be a useful way to 
use geometric enhancement to improve the communication of spatial information to a viewer. The 
distortion can be used to correct underlying natural biases in spatial judgements. For example, 
exocentric direction judgements (Howard, 1982) made of extended objects in perspective displays 
can, for some response measures, exhibit a "telephoto bias." That is to say, the subjects behave as 
l they were looking at the display through a telephoto lens. This bias can be corrected by intro- 
duction ° faCOmpensating wid e-angle distortion (McGreevy and Ellis, 1986; Grunwald and Ellis, 


SYMBOLIC ENHANCEMENT 


Symbolic enhancements generally consist of objects, scales, or metrics that are introduced into 
a display to assist pick-up of the communicated information. The usefulness of such symbolic aids 
can be seen, for example, in displays to present air traffic situation information which focus atten- 
tion on the relevant variables" of a traffic encounter, such as an intruder's relative position, as 
opposed to less useful "properties" of the aircraft state, such as absolute position (Falzon, 1982). 

One way to present an aircraft's position relative to a pilot's own ship on a perspective display 
is to draw a grid at a fixed altitude below an aircraft symbol and drop reference lines from the sym- 
bol onto the grid (fig. 20). If all the displayed aircraft are given predictor vectors that show future 
position, a similar second reference line can be dropped from the ends of the predictor lines. 

The second reference line not only serves to clearly show the aircraft the future position of the 
aircraft on the grid, but additionally clarifies the symbol’s otherwise ambiguous aspect. Inter- 
estingly. it can also improve perception of the target's heading difference with a pilot's ownship. 
This effect has been shown in an experiment examining the effects of reference lines on egocentric 
perception of azimuth (Ellis, Grunwald, and Velger, 1987). I wish to briefly use this experiment 
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as an example of how psychophysical evaluation of images can help improve their information dis- 
play effectiveness. 

In this experiment subjects viewed static perspective projects of aircraft-like symbols elevated 
at three different levels above a ground reference grid: a low level below the view vector, a middle 
level colinear with the viewing vector, and a high level above the view vector. The aircraft sym- 
bols had straight predictor vectors projecting forward, showing future position. In one condition, 
reference lines were dropped only from the current aircraft position; in the second, condition lines 
were dropped from both current and predicted position. 

The first result of the experiment was that subjects made substantial errors in their estimation of 
the azimuth rotation of the aircraft; they generally saw it rotated more towards their frontal plane 
than it in fact was. The second result was that the error towards the frontal plane for the symbols 
with one reference line increased as the height of the symbol increased above the grid. Most sig- 
nificantly, however, introduction of the second reference line totally eliminated the effect of height, 
reducing the azimuth error in some cases almost 50% (fig. 21). 

More detailed discussion of this result is beyond the scope of this talk; however, these experi- 
mental results show in a concrete way how appropriately chosen symbolic enhancements can pro- 
vide not only qualitative, but quantitative, improvement in pictorial communication. They also 
show that appropriate psychophysical investigations can help designers define their spatial 
instruments. 


COMBINED GEOMETRIC AND SYMBOLIC ENHANCEMENTS 


Some enhancements combine both symbolic and geometric elements. One interesting example 
is provided by techniques connecting the photometric properties of objects or regions in the display 
with other geometric properties of the objects or regions themselves. Russell and Miles (1987) 

(see also Paper 48), for example, have controlled the transparency of points in space with the 
gradient of the density of a distributed component and produced striking visualization of 3D objects 
otherwise unavailable. These techniques have been applied to data derived from sequences of MRI 
or CAT scans and allowed a kind of "electronic dissection" of medical images. Though these 
techniques can provide absolutely remarkable images, one of the challenges of their use is the 
introduction of metrical aids to allow the viewer to pick up quantitative information from the 
photometric transformation (Meagher, 1985, 1987). 


COMPUTATIONAL ENHANCEMENTS 


While considerable computation may be involved in the rendering and shading of static pic- 
tures, the importance of computational enhancement is also particularly evident for shaping and 
placing objects in interactive spatial instruments. In principle, if unlimited computational resources 
were available, no computational enhancements would be needed. The enhancements are neces- 
sary because resources must be allocated to ensure that the image is computed in a timely and 
appropriate manner. 
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An example of a computational enhancement can be found in the selection of a type of geomet- 
ric distortion to use as a geometric enhancement in a head-mounted, virtual-image computer display 
of the type pioneered by Ivan Sutherland (1970) (fig. 22). Distortions in the imagery used by such 
displays can be quite useful, since they are one way that the prominence of the components of the 
image could be controlled. 

It is essential, however, that the enhancements operate on the displayed objects before the 
viewing transformation, because, here the picture of reality collides with a picture for commu- 
nication. The virtual-image presentation makes the picture appear in some ways like a real space. 
Accordingly, distorting geometric enhancements that are computed after the viewing transformation 
can disturb visual-vestibular coordination and produce nausea and disorientation. This disturbance 
shows how different computational constraints distinguish head-mounted from panel-mounted 
formats. 

A second example of a computational enhancement is shown on the interactive, proximity- 
operations, orbital planning tool developed by Art Grunwald in our laboratory. When first imple- 
mented, the user was given control of the direction and magnitude of the thrust vector; these 
seemed reasonable, since they are the basic inputs to making an orbital change. The nonlinearities 
and counterintuitive nature of the dynamics, however, made manual control of a predictor cursor 
driven by these variables impossible. The computational trick needed to make the display tool 
work was allowing the user to command that the craft be at a certain location at a set time and allow 
the computer to calculate the required bums through an inverse orbital dynamics algorithm. This 
technique provided a good match between the human user's planning abilities and the computer's 
massive computational capacity. 

A third example of a computational enhancement is shown on the same interactive, proximity- 
operations, orbital planning tool. Despite the fact that the system has been implemented on a high- 
performance 68020 workstation with floating-point processor and dedicated graphics geometry 
engine, unworkably long delays would occur if the orbital dynamics were constantly updated while 
the user adjusted the cursor to plan a new way-point. Accordingly, the dynamics calculations are 
partially inhibited whenever the cursor is in motion. This feature allows a faster update when the 
user is setting a way-point position and eliminates what would otherwise be an annoying delay of 
about 0.3 sec while adjusting the way-point position. 

When Arthur Grunwald finished the first iteration of this display, we decided to name it. Like 
a dutiful NASA researcher, he searched for a acronym-something like Integrated Orbital and 
Proximity Planning Systems, or IOPPS for short. This looked to me like it might sound like 
OOPS and I thought we should find a better name. I asked him to find maybe a Hebrew name that 
would be appropriate. He thought about it for awhile and came up with Navie, or "reliable 
prophet." This is perfect, since that is exactly what the display is intended to provide: reliable 
prophesy of future position. 

But there is another sense in which Navie is a good name. I would like to think that it, and 
other display concepts developed in our division and elsewhere, also provide a kind of prophesy 
for the coming displays to be used by NASA during future unmanned, and manned, exploration of 
air and space. 
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Like most human activities, this exploration is not an endeavor that can be automated; it will 
require iteration, trial and error, interactive communication between men and machines a™ 
between men and other men. The media for this communication must be designed. Some ol them 
will be spatial instruments. 
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Figure 1 Prehistoric cave painting of animals from southwestern France. 



Figure 2.- Woodcut by Diirer illustrating how to plot lines of sight with string in order to make a 
correct perspective projective. 
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Figure 3 - Egyptian hieroglyphic for the Eye of Horns illustrating the symbolic aspect of picto- 
graphs. Each part of the eye is also a symbol for a commonly used fraction. These assign- 
ments follow from a myth in which the Sun, represented by the eye, was tom to pieces by the 
God of Darkness later to be reassembled by Thoth, the God of Learning. 



Figure 4.- Leonardo’s sketch of two hands using shading to depict depth. 
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Figure 5- Crivelli's Annunciation illustrating strong perspective convergence associated with 
wide-angle views that can exaggerate the range of depth perceived in a picture. 
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Figure 6.- An engraving by Escher illustrating how the ambiguity of depicted height and depicted 
depth can be used in a picture to create an impossible structure, apparently allowing water to 
run uphill. © 1988 M. C. Escher heirs/Cordon Art-Baam-Holland. 
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Figure 7.- Urban freeways, a painting by Thiebaud showing an instant of time on a California 
freeway. 
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Figure 8- View of the Prague town hall clock, which indicates the positions of heavenly bodies as 
well as the time. 
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Figure 9.- Fragments of an ancient Greek mechanical device used to calculate the display positions 
of heavenly bodies. 
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Figure 10.- An old map of the world from the 17th Century. 
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Figure 1 1- Rhumb-line and great-circle routes between two points on the globe. Note the con- 
stant bearing of the rhumb-line route and the constandy changing bearing of the great-circle 
route. On the globe the great-circle route is analogous to a straight line and direction Z is the 
azimuth of B from A. 
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Figure 12.- Plate caree projection illustrating the curved path traced by a rhumb line on this format 
i.e., line AEFG. 
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Figure 13.- Mercator projection illustrating how a nonlinear distortion of the latitude scale can be 
used to straighten out the path traced by a rhumb line. 



Figure 14 - Muybridge’s photographic sequence of a goat walking. The background grid pro- 
vides a reference for measuring the pattern of limb movement. 


1-19 



ORIGINAL PAGE 

BLACK AND WHITE PHOTOGRAPH 



Figure 15. View of the forward panel of a 737 cockpit showing the artificial horizon on the atti- 
tude direction indicator. 



Figure 16.- An advanced-concepts commercial aircraft cockpit in the Man- Vehicle Systems 

Research Facility of NASA Ames Research Center. This artist’s conception shows how future 
cockpits may resemble ordinary offices. 
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Figure 17 - Sample view from an interactive-graphics-based, planning tool to be used in assisting 
informal changes in orbits and proximity operations in the vicinity of a space station. 



Figure 18 - Possible display format for a commercial aircraft cockpit traffic display. The pilot s 
own craft is shown in the center of the display. All aircraft have predictor vectors attached 
showing future position and have reference lines to indicate height above a reference grid. 




z 

Figure 19 - Illustration of the geometry of perspective projection showing the azimuth and the 
elevation of the viewing vector InR, directed from the center of projection COP. 








Figure 20.- Five views of sample stimuli used to examine the perceptual effect of raising an air- 
craft symbol above a reference grid. The attitude of the symbol is kept constant. Addition of a 
second vertical reference line is seen to reduce the illusory rotation caused by the increasing 
height above of the grid. 
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CCW MEAN DIRECTION ERROR, deg 



Figure 21- Mean clockwise and counterclockwise egocentric direction judgement for clockwise 
azimuth rotation of an aircraft symbol. 



Figure 22 - Probably the first computer-driven head-mounted viewing device. It was developed 
by Ivan Sutherland to give the viewer the illusion of actually being in the synthetic world 
defined in the computer. 
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